Do we really want to call the initializer multiple times or should the second import become a no-op?

I think the spirit of PEP 489 (which we don't support yet, but hopefully are heading towards) is that modules can be torn down and re-imported, so I think it's correct to call the initializer multiple times.

which we don't support yet, but hopefully are heading towards

I think this is the main issue here, isn't it? Does this change mean we opt into allowing tear-down and reinitialization of PyO3-based modules? What invariants would this break? Do we test this somewhere?

(I am not trying to say that it doesn't work. Just that I myself do not understand the consequences of this change and have little confidence about its effects on existing code.)

In pydantic/pydantic#6584 (comment), you mention opt-in. Might this be advisable even for this limited relaxation? What if the module is torn down but static data still contains pointers into the old module (e.g. capsules) which are now invalid? For example, is the code in https://github.com/PyO3/rust-numpy/blob/c16fbb1e630b0538fdf17cbc53043bf0467459f9/src/npyffi/mod.rs#L30-L32 valid after this change?

So I found the Cython implementation here:

https://github.com/cython/cython/blob/ce1aa59ab6300e18e50c752b5c9e28a04b665060/Cython/Utility/ModuleSetupCode.c#L1648C18-L1648C18

That code:

• Has a similar check that the interpreter hasn't changed ID
• Caches a module instance in a C static and returns the single module instance always.

So I guess let's implement this strategy too and defer the question of PEP 489 compatibility to a future PR. My goal for this PR is primarily to fix the use cases in the OP, which I believe the Cython strategy would cover.

It's clearly awkward to get this right. I think on balance it's desirable if we match Cython so that more of the ecosystem fails in the same way and it seems better that we only ever run the module initializer once for now. We can make progress towards per-module state to eventually remove this worry.

I agree with that. Do you know the upstream position on this, i.e. would they consider fixing this or does it work as intended?

(I made with_embedded_python_interpreter unsafe for this reason.)

I am somewhat confused as to what the consequences of this discovery are. Especially with the Cython module immediately showing the failure mode I was concerned about. Can we still relax this? Is the unsafe on with_embedded_python_interpreter sufficient? (I guess another other way to re-initialize Python in the current process would call unsafe Rust code or code in inherently unsafe languages like C, i.e. is trusted?)

I have just posted python/cpython#109785 - let's see what the outcome of that discussion is.

No movement yet on the upstream issue. @adamreichold are you happy if I proceed to merge this? As above at least we'll then be consistent with Cython and I think the generally accepted position is that things are expected to be wonky after finalizing and re-initializing the interpreter.

Yes, I think the language on with_embedded_python_interpreter is strong enough. This is the only API we provide to finalize and then re-initialize the interpreter?

Yes (apart from the raw FFI bindings). auto-initialize never bothers finalizing the interpreter.